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ABSTRACT 

Accepted guidelines for academic program review often 
clash with the needs of administrators, who require timely and 
frequent information foi monitoring the quality of academic programs. 
A method is described for developing key indicators that permit 
annual scanning of all programs. The goal was to develop indicators 
that would flag programs showing unusual variation; these programs 
are then selected for more thorough, focused evaluation. The process 
described developed the following model components, with specified 
program characteristics for each: (1) market demand (student demand, 
employer/business demand, demand from transfer institutions); (2) 
resources (students' entry skills, faculty/staff, space, equipment, 
supplies, operating budget, and college facilities and services); (3) 
process (applications, enrollments, academic progress and 
performance, full time/part time faculty ratios, student 
satisfaction, retention, withdrawals, and changes of major); and (4) 
outcomes (graduation, placement, employment, transfer rates, 
achievement of competencies, salaries, "transferability," and 
satisfaction of students, employers and transfer institutions). 
Validation of the model is also discussed. Contains 10 references. 
(Author/KM) 
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Abstract 

Accepted guidelines for academic program review often clash 
with the nee.ds of administrators, who require timely and frequent 
information for monitoring the quality of academic programs. This 
paper describes a method for developing key indicators that permit 
annual scanning of all programs. The goal was to develop 
indicators that would flag programs showing xinusual variation; 
these programs are then selected for more thorough, focused 
evaluation. The intended audience is institutional researchers 
involved in the academic program review process. 



Introduction 

An extensive literature axists on models, methods, and 
principles for conducting academic program reviews (cf. Barak, 
1982; Conrad & Wilson, 1985). In practice, however, guidelines 
for thorough and accurate data collection procedures often clash 
with the needs of administrators who use the information. For 
example, a sound program review is comprehensive (examining all 
facets of a program) , uses multiple criteria to assess each 
program facet, is timely, and presents data in a summary form that 
is easy for administrators to use (Barak, 1982; Conrad & Wilson, 
1985; Mark & Shetland, 1987). 

The requirements of comprehensiveness and timeliness are 
especially difficult to balance. Moreover, the operational 
definition of "timeliness" is changing. In the past, a full- 
fledged audit of each program every 5 or 6 years was a common 
accreditation standard, but the standard is shifting to ongoing, 
continual evaluation (Middle states Association of Colleges and 
Schools, 1982). This shift, which reflects the quickening pace of 
change in our society, has generated a need for supplementary 
annual scanning of all programs (Barak, 1982) . 

This paper describes a method for developing a set of key 
program indicators permitting annual scanning across all academic 
programs. The goal was to develop indicators that would flag 
programs showing unusual variation; these programs would be 
selected for more thorough, focused review. 



LiwjtatjQns fif tlie Existing Review Process 

At Delaware County Community College, the existing policy on 
academic program audits was developed in 1980. The policy 
requires that each academic program be given a full scale audit at 
least once every five years. New programs are to be audited 
within two years of their start-up date. The Office of the Vice- 
President for Instruction identifies the programs to be audited in 
a given year. An evaluation team, which is formed for each audit, 
outlines key questions to be addressed in the audit. 

The Research Office is responsible for data collection, data 
analysis, and reporting. The report prepared by the Research 
Office is reviewed in turn by the Evaluation Team, the Vice- 
President of Instruction, the President, and the Board of 
Trustees. Each of these reviewers makes comments and recommenda- 
tions before sending the report to the next reviewer. The 
President recommends to the Board of Trustees action on the 
program ("continue, revise, or phase out") . 

The audit report is comprehensive. It includes all program 
and course competencies and their relationship to one another, 
staffing patterns, faculty background and experience, students' 
evaluation of instruction, historical analysis of course 
scheduling patterns, faculty evaluation of the program, an 
inventory of facilities, equipment, and support services, student 
and faculty evaluation of facilities, equipment, and support 
services, student profiles, enrollment trends, retention rates, 
students' academic performance and progress, discipline and 
program costs, regional and local labor market supply and c?r^mand. 



enrollments and graduates at other local colleges, placement rates 
and salaries of graduates, and graduates' and employers' 
assessments of the program. 

Conducting an audit requires a major investment of the 
Research Office's resources, and several serious flaws in the 
process have become evident. The goal of auditing programs on a 
five-year cycle, with new programs receiving an audit within two 
years, has been unattainable, despite efforts to combine 
"clusters" of related programs into one audit. The problem has 
been exacerbated by the proliferation of new curricula offered by 
the college during the past decade. The time required to complete 
the audit cycle (a minimum of 1 year to prepare the initial 
report, another year or two to complete the review cycle) often 
results in ^ "stale" report. By the time the Board of Trustees 
reviews the audit report, a program may have undergone substantial 
change and may bear little resemblance to the curriculum described 
in the report. 

Despite these flaws, the audits have had value; most audits 
yield several major findings that identify areas of strength 
and/or weakness. However, the effort expended in conducting the 
audit is disproportionate to the results it yields. What is 
needed is a more efficient method of identifying a program's 
essential strengths and weaknesses. 

As the limitations in the audit process were becoming evident, 
the student demographics at DCCC and the mix of programs and 
services were changing rapidly. These trends emphasized the need 
for a method of monitoring all programs, on an annual basis if 
possible, to help administrators track the changes occurring 



within academic programs. An annual monitoring mechanism could be 
used to identify programs requiring more thorough evaluation- 
These evaluations of targeted programs would be focused on the 
areas of concern. This approach would reverse the existing audit 
procedure, in which every program is subjected to a full scale, 
comprehensive audit, even though questions about the program might 
be restricted to selected features of the program. 

Method 

In 1987 the college was awarded a grant from the Metropolitan 
Life Foundation to develop a Key Indicators model for curriculum 
review. The Key Indicators Project was staffed by the Vice- 
President for Instruction, the Dean of Enrollment Management, 
Research, and Planning, the Director of Institutional Research, 
and the Registrar. The project team's first step was to gather 
information on all program facets that might yield relevant 
indicators. The tearo members conducted brainstorming sessions to 
generate ideas. Meetings were held with the college deans and 
assistant deans, to identify the features they take into consider-* 
ation in assessing the quality of the programs they administer. A 
literature review was conducted, and the other community colleges 
in the state were surveyed to determine their procedures for 
evaluating academic programs. 
Developing the Model 

To manage the large amount of information generated by these 
efforts, the team developed a model, borrowed from Elkin & 
Molitor's (1984) guidelines for developing management indicators. 
The model contains four basic components: market demand, 
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resources, process, and outcomes. Most ideas generated from the 
brainstorming session, the meetings with administrators, the 
statewide survey, and the literature search fit into one of these 
categories. 

The next step was to list, for each model component, existing 
data sources that might serve as measures of that component of the 
model. The team agreed that the pilot set of indicators should 
rely upon data al*:eady available or accessible. As the key 
indicators are tested and refined, new data collection methods 
would be developed to fill in gaps that appear in the model • 

The Key Indicators Project Team compared the "wish list" of 
measures with the list of currently available data sources; the 
team identified characteristics of academic programs that met two 
criteria: 1) characteristics frequently cited by the team and by 
other administrators as important indicators, and 2) characteris- 
tics for which some measure was available. The characteristics 
most often cited are shown in Table 1. Table 2 shows data sources 
that currently exist to measure these characteristics. 
Total Qualitv Philosophy; Basis for Defining Indicators 

As the areas to be targeted for monitoring began to 
crystallize, the team started to work on operational definitions 
of the indicators. The team's approach to the task was influenced 
by concepts in the Total Quality literature (cf. Deming, 1986; 
Imai, 1986) and by methods described in the statistical process 
control literature (e.g. , Grant & Leavenworth, 1974 ; Ishikawa, 
1976) . A major tenet of this literature is that one must gather 
data on processes to determine their unique "fingerprint" or 
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TABLE 1. 

Program Characteristics Most Commonly Cited 
as Important Indicators 

Model component Procirajd Characteristics 

Market Demand Student Demand, Employer/Business Demand, 

Demand from Transfer Institutions 

Resources Students' Entry Skills 

Faculty/Staff (Availability, Expertise) 
Space , Equipment , Supplies , Operating 
Budget 

College Facilities and Services 

Process Applications, Enrollments 

Academic Progress and Performance 
Full Time/Part Time Faculty Ratios 
Student Satisfaction 

Retention, Withdrawals, Changes of Major 

Outcomes Graduation, Placement, Employment, 

Transfer Rates 
Achievement of Competencies 
Salaries 

"Transferability" 

Satisfaction of Students, Employers, 
Transfer Institutions 
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TABLE 2 

Existing Data Sources for Each Model Component 
Model Component 

I. Market Demand (Student Demand, Employer, Transfer Institution) 

1) Number of new students 

2) Number of applications 

3) Proxy measure: Percent employed in related position (see 
outcomes component) 

4) Proxy measure: Percent continuing education (see outcomes 
component) 

II. Resources (Program, Faculty/Staff, Facilities/Services, 
Students) 

5) Proportion of credit hours (across the. fiscal year) in 
core courses taught by full-time faculty 

6) Ratio of: Direct cost of program to program revenue 

7) Capital equipment cost for discipline (where discipline- 
program match exists) 

8) Proportion of entering students scoring at the Basic or 
Developmental level. 

III. Academic Performance 

9) New students as proportion of total in major 

10) Term-to-term retention: all students 

11) Term-to-term retention: new students 

12) Graduation rates 
IS) GPA 
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Table 2 (Continued) 

14) Proportion of credits successfully completed* 

15) Success rates in gate courses (gate courses identified 
by assistant deans) 

16) Proportion of second-year students 

17) Proportion enrolled in courses in own discipline 
(applies to selected programs) 

( — ) Indicator to be developed: proportion of core courses 
successfully completed 
IV • Outcomes 

18) Placement rates 

19) Proportion employed in related field 

20) Proportion transferring 

21) Median full time salary 

22) Graduates' evaluation of program 

23) Employers' evaluation of program 



- This indicator was dropped after initial testing. 
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pattern of variation. Studying processes in this way enables the 
researcher to distinguish between normal, random variation in the 
process (variation due to "common causes") c 1 variation signaling 
that something has; occurred (variation due to "special causes") • 

The team's goal was to gather trend data for each area 
targeted for monitoring and then to devise criteria for flagging 
programs showing unusua] variation. Each program's past 
performance would be used as a baseline to define the "normal" 
range of variation for that program. 

The team's approach adopted some methods commonly used to 
develop key indicators and rejected others. One common feature of 
the indicators described in the literature is their reliance on 
recent trends as a basic data collection measure. For example, 
many indicators cited in Elkin & Molitor's (1984; description of 
management indicators and in the Columbus Technical Institute's 
model of program review (Task Force on Program Review and 
Evaluation, 1982) use the most recent 3-year trend as baseline 
data. The Key Indicators team was committed to using trend data, 
but wished to empirically test different time intervals to 
determine which worked best for Delaware County Community College. 

A common feature that was rejected by the team was the us«i of 
a standard that generates inter-program comparisons. The reassons 
for rejecting this approach are reflected in Deming's (1986) crit- 
icisms of "management by objectives" and "management by numbers". 
The basic flaw lies not with the use of standards per se, but with 
managers' tendency to apply standards that are irrelevant to the 
process and to customer needs. Imai (1986) makes a similar point 




in his distinction between "results-oriented management" and 
"process-oriented management" . 

Managers tend to grasp any available or convenient benchmark 
and use it as a standard, regardless of its appropriateness to the 
processes or prograias being compared. Whether the focus is 
enrollments, placement rates, or retention rates, administrators 
attempting to interpret data will seek a basis for comparison, 
such as the college-wide r.ate. But programs may depart 
significantly from the college-wide rate for reasons unrelated to 
program quality. Differences among programs often reflect valid 
differences in program structure and goals. 

In the early phases of developing the Key Indicators, the 
team repeatedly fell into this trap of evaluating programs by 
comparing their performance to a college-wide or divisional 
standard. The team freed itself from this trap by borrowing 
Quality Improvement statistical process control concepts most 
often used in manufacturing settings (Ford Motor Co., 1983). One 
principle is that no attempt should be made to improve or change a 
process until the process has been studied to determine if it is 
stable. A process is said to be stable (or "in statistical 
control") if variations in the process are due solely to random 
variation; in other words, no variation is due to "special 
causes" . 

In adapting this philosophy to our Key Indicators model, the 
team agreed that the purpose of laach indicator would be to examine 
sach program with its past performance and to set flags that would 
indicate that an unusual change ("specifil cause") had occurred. 
The flag conveys no direct information about program quality; 

11 

, X4 



change that has occurred may be favorable, unfavorable, or 
irrelevant • The flag serves solely to point managers to programs 
that require closer scrutiny* A well-developed set of indicators 
will also suggest which features of a program should be targeted 
for closer study. 

This approach to program evaluation is a radical departure 
from the previous approach taken at Delaware County Community 
College, in which a full scale audit was applied to every program 
selected for audit. The Key Indicators approach is a preventive 
one that strives for early warning of potential problems. The 
previous approach was often a remedial one; by the time a program 
had been selected for evaluation, it could have developed serious 
problems. 

Developing Indicator Flags 

By June, 1988, the basic philosophy of the Key Indicator 
team s approach had been agreed upon, the model had been 
clarified, and the team began testing indicators and flagging 
systems. Table 3 summarizes the approaches tested. To test each 
approach, the team applied the indicator across all programs to 
determine which programs would have been flagged if the indicator 
had been in use in 1987-88. Hindsight gave us, the opportunity to 
assess whether the flagging would have been appropriate. 

Method 1 flagged any program showing a continuing trend (over 
3, 4, or 5 years) on en indicator. Three-year trends turned out 
to be a common pattern, too common to be useful; four-year and 
five-year trends, on the other hand, were too unusual to be 
useful. 
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TABLE 3 

Methods Tested for Setting Flags 



on Key Indicators 



METHOD 

!• Flag programs showing a trend 
(3-, 4-, 5-year) in the abso- 
lute value of an indicator. 

2* Flag programs falling outside 
limits defined by recent his- 
tory of program: 3 -year mov- 
ing average plus or minus 
standard deviation based on 
5-year history 

3. Flag program based on the 
degree to which it departs 
from projections, with pro- 
jections based on either: 

a) moving average or 

b) regression line based 
program history 



FINDING 

3 - year trends too common; 

4 - year and 5 -year trends 
too rare 

Major limitation: no 
correction for college- 
wide trends 



(a) and (b) generated 
similar projections, with 
(a) yielding slightly 
better prediction of 1988 
data. Method 3(a) chosen 
as flag for pilot set 
of indicators. 
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Method 2 flagged programs falling beyond the limits of the 
program's historical variation. We tested different moving- 
average intervals (3, 4, and 5 years) and different definitions of 
limits. The major drawback to this approach is that it does not 
correct for a college-wide trend that affects all programs 
equally. 

Method 3 flagged programs with a Key Indicator value that 
departed significantly from pro j ections based on the program ' s 
recent histoiry. Method 3^ like Method 2, flagged programs with a 
changing performance on an indicator. Method 3, however, 
correc' s for generic trends; it flags programs with discrepancy 
scores that depart from the norm for the current year. Thus, a 
sudden rise in enrollments that affects many programs in a 
particular year will not result in all programs being flagged. 
Only programs showing change departing from the norm are flagged. 
This approach, of course, does incorporate into the model a 
college-wide standard. However, the standard is used as a measure 
of stability, not as an evaluative measure. 

After defining the indicator as thr- discrepancy between the 3- 
year moving average and the current value of an indicator, the 
team tackled the task of choosing a flagging system. Once again, 
t" B team tested different parameters and evaluated the results 
using current knowledge about the programs as a validity test. 
The 1-standard deviation limit was chosen because it flagged 
programs that, according to hindsight, deserved a closer look. 

Table 4 shows the Fall-to-Winter retention Key Indicator (#10 
on Table 2) . Programs are flagged if their discrepancy score 
("Year 4 minus 3-Year Mean") falls outside the limits defined by 
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TABLE 4 

Key Indicator Example: Fall-to-Winter Retention Rates 



PALL-TO' 


-WINTER RETENTION 




BY PROGRAM 




PROGRAM 


VEAP 1 




VP AO ^ 




VPSD A 


XtlAR 4 










MEAN 




MINUS . 












3- 


-YEAR ME 


PERIOPERATIVE 


9 


33 




21 


42 




CLERICAL 


49 


42 


J / 


43 


60 


1 7 


BIOMEDICAL EQUIP TECH 


75 


79 




74 


89 




RESPIRATORY THERAPY 


94 


90 




89 


100 




CHEMICAL TECH 


55 


71 




63 


71 


Q 
0 


MICROCOMPUTERS 


28 


21 


Oft 


26 


32 


£i 


FIRE SCIENCE 


66 


56 


iO 


66 


70 


A 


ELECTRONICS 


66 


72 


OO 


69 


72 


•) 


TECHNICAL STUDIES 


67 


41 


/ 0 


61 


64 




UNIT CLERK 


64 


40 


62 


55 


58 


<^ 


EARLY CHILD ED 


72 


77 


•7A 


74 


76 




GENERAL ED 


53 


49 


Afi 
*kO 


50 


51 


X 


LEGAL SECRETARY 


75 


74 


Ox 


70 


70 




NATURAL SCIENCE 


67 


71 


67 
0 / 


68 


68 




ADMINISTRATIVE SEC 


73 


62 


0 / 


67 


67 




LIBERAL ARTS 


65 


59 


6*^ 


62 


62 


0 


DRAFTING/DESIGN 


48 


62 


72 


61 


59 


-2 


COMPUTER SERV TECH 


74 


74 


71 


73 


71 




ENGINEERING 


70 


66 


74 


70 


68 


4b 


MEDICAL SECRETARY 


80 


63 


73 


72 


70 


-2 

4w 


BUSINESS ADMIN 


71 


67 


67 


68 


66 


-2 

4w 


COMPUTER OPERATIONS 


74 


66 


65 


68 


66 


-2 

4b 


BUSINESS MANAGEMENT 


64 


64 


68 


65 


63 


-2 


SECRETARIAL CERT. 


46 


47 


49 


47 


45 


-2 


ADMIN OF JUSTICE 


69 


72 


74 


72 


68 


«*4 


ACCOUNTING 


64 


64 


60 


63 


58 




COMM/GRAPHIC ART 


84 


74 


65 


74 


69 


-5 


NURSING 


94 


96 


88 


93 


87 


-6 


RETAIL MANAGEMENT 


81 


78 


72 


77 


71 


-6 


HOTEL/REST MANAGEMENT 


72 


74 


70 


72 


65 




MEDICAL ASSISTANT 


78 


80 


77 


78 


71 


-7 


CLIMATE CONTROL 


73 


80 


67 


73 


64 




SURGICAL TECH 


82 


88 


87 


86 


75 


-11 


ROBOTICS 


100 


72 


68 


80 


69 


-11 


COMPUTER SCIENCE 


71 


75 


75 


74 


62 


-12 


COMPUTER PROGRAMMING 


59 


57 


55 


57 


45 


-12 


CONSTRUCTION TECH 


54 


63 


59 


59 


45 


-14 


MECHANICAL TECH 


71 


45 


57 


58 


41 


-17 


ENERGY SYSTEMS 


56 


86 


83 


75 


35 


-40 



MEAN 

STANDARD DEVIATION 



-2 
10 
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. the mean discrepancy (-2) plus or minus one standard deviation 

unit (S.D. - 10). Even though a retention report is issued each term, 
showing historical retention rates for each program, the pattern 
shoivn-by the "outlier" programs in the table is not evident in 
that report. The team's assessment of this indicator was that it 
revealed patterns that do not emerge from our typical analysis of 
retention rates. It should be noted that the Key Indicators team 
tested many variations on Method 3 before settling on the method 
illustrated in Table 4. Method 3 will not necessarily be the 
optimal approach for other institutions. 

- Different flags will have different program implications. For 
example, new programs should set off flags in the early years of 
operation, as enrollments build. The team found these patterns to 
be useful as validity tests of the Key Indicator flagging system. 

Once the method for flagging programs was chosen for the first 
indicator tested, the team found that the approach could be 
generalized to the other characteristics chosen for the pilot set 
of indicators. Limitations in the availability of data forced the 
team to use a 2 -year average for several program characteristics 
in the model. However. 3-year trends will be available in the 
1989-90 year for all progrem characteristics. 
The Key Indicator Outcome Matrix 

As the team completed the transformation of program charac- 
teristics into actual indicators, it became apparent that a method 
of sxommarizing the results across the entire model was necessary. 
The plan is to generate a matrix similar to the one shown in Table 
5. With experience, this matrix should help us to interpret 
different patterns or clusters of flags and to understand their 
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TABLE 5 

Key Indicator SuMMry Katrix 
MODEL COMPONENT 





I. H 


AR\E 


T DEI 


4AND 




II. REFERENCES 




III. ACADEMIC PROGRESS 


1 


IV. OUTCOIE 


1 


2 


3 




5 


6 


7 


8 


9 


10 


11 


12 


13 


14 


15 


16 


17 


! 18 




20 


21 


[ 22 


23 


Accounting 






















































Acfti of Justice 






















































Etc* 























































A will be entered in a cell when the curriculum was flagged on the high side of the indicator. 
A will be entered in a cell when the curriculum was flagged on the low side of ths indicator. 



implications for program quality. The matrix should also aid in 
eliminating redundant and obsolete indicators, and in identifying 
gaps in tiiJi^ model. The model can work only if the indicators are 
reviewed and modified continually, to keep it efficient and 
responsive to the needs of those who use it. 

Validating £bS KS.Y indicators Model 
One of the first tests of the model will be to compare and 
contrast the information yielded by the pilot set of indicators 
with the results of an audit that has been conducted over the past 
year on a cluster of four academic programs. To be successful, 
the Key indicator model should replicate the major findings of the 
audit. Preliminary indications are that the Key indicators model 
will identify patterns of change, and chat these patterns will be 
more apparent from the Key indica '-ors model than from the audit 
report. 

As noted above, the model will require continual refinement 
and modification if it is to keep pace with managers' needs for 
information. The Key indicators te?ja has already identified indi- 
cators that ard missing from the model beoaUoe the data are not 
currently available^ For example, tha college lacks data on 
transfer institutions' assessments of our graduates' competencies. 
Also lacking is extensive data on students who leave the college 
without graduating. In the next round of reviewing the model, we 
will attempt to generate the data needed to fill these gaps. 
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FOOTNOTES 

1) This project was funded through ths Nstroirolitan Life Founda- 
tion Community College Awards for Institutional self-study. 
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